Systems, methods and apparatus of handling structures in three-dimensional images

ABSTRACT

Systems, methods and apparatus are provided through which in some embodiments, a structure manager explicitly creates a container of graphical objects of anatomical regions by adding a structure, or the structure manager implicitly creates graphical objects of a group of anatomical regions through an organ segmentation process.

RELATED APPLICATION

This application is related to copending U.S. application Ser. No.10/858,241, filed Jun. 1, 2004 and titled “Systems and Methods forSegmenting an Organ in a Plurality of Images.”

This application is related to copending U.S. application Ser. No.10/935,893, filed Sep. 8, 2004 and titled “Contrast Agent Imaging-DrivenHealth Care System and Method.”

This application is related to copending U.S. application Ser. No.10/907,690, filed Apr. 12, 2005 and titled “Method and System forAutomatically Segmenting Organs from Three Dimensional ComputedTomography Images.”

This application is related to copending U.S. application Ser. No.11/352,477, filed Feb. 11, 2006 and titled “SYSTEMS, METHODS ANDAPPARATUS OF HANDLING STRUCTURES IN THREE-DIMENSIONAL IMAGES HAVINGMULTIPLE MODALITIES AND MULTIPLE PHASES”

FIELD OF THE INVENTION

This invention relates generally to imaging systems, and moreparticularly to three dimensional imaging systems.

BACKGROUND OF THE INVENTION

Images of a structure of an object are generated in one of a number ofconventional modalities. In medical care, where the object is a patient,the images are suitable for diagnostic purposes or radiotherapytreatment, or for planning surgery.

Examples of the conventional modalities include conventional X-ray planefilm radiography, computed tomography (CT) imaging, magnetic resonanceimaging (MRI), and nuclear medicine imaging techniques, such as positronemission tomography (PET) and single photon emission computed tomography(SPECT).

A three-dimensional (3D) medical image is a collection of adjacent(transaxial) two-dimensional (2D) slices. Clinicians recombineanatomical elements of 2D slices to form a 3D image of an anatomicalregion or an organ. This recombination process is usually termedreconstruction.

During clinical diagnosis, the patient's internal anatomy is imaged todetermine how a disease has progressed. The infected tissues show somedifferences from normal tissues. Also, the patient may have some type ofindividual differences or abnormalities regarding healthy tissues.

The clinicians identify and handle critical anatomical regions, and inparticular organs, on several images for planning of treatment orsurgery. Handling critical anatomical regions and organs includestracing the outline of these regions and organs, which yields graphicalobjects. A graphical object visually marks for the clinician theseparation of an anatomical region from the other parts of an image.Manually drawing the individual contours on a contiguous set of 2Dslices then combining them is very time consuming and labor intensive.The time and labor increases significantly with the number of imageslices, the number and sizes of the organs, tumors, etc. in theanatomical area of interest. The quality of the contouring and 3D visualgraphical objects generated from the 2D slices depends on the resolutionand contrast of the 2D images, and on the knowledge and judgment of theclinician performing the reconstruction. However, conventional methodsof segmentation of anatomical regions and organs by the clinicianrequire a considerable amount of time to be performed and subjectivityin the judgment of the clinician in manual segmentation introduces ahigh degree of imprecision.

The graphical objects also need to be managed. Conventional methods ofmanaging the graphical objects are inefficient and overwhelming to theabstraction skills of human clinicians.

For the reasons stated above, and for other reasons stated below whichwill become apparent to those skilled in the art upon reading andunderstanding the present specification, there is a need in the art formore efficient methods and apparatus of managing graphical objects.There is also need to reduce the time and imprecision of humanclinicians in segmenting anatomical regions.

BRIEF DESCRIPTION OF THE INVENTION

The above-mentioned shortcomings, disadvantages and problems areaddressed herein, which will be understood by reading and studying thefollowing specification.

The systems, method and apparatus described below are a complex system,yet are an efficient and user-friendly system, which managesorganization of graphical objects and manual/automated contouring(segmentation). The systems, method and apparatus described below aresuitable for any kind of image modality and any kind of segmentationalgorithm.

In one aspect management of the graphical objects includes grouping theobjects together for classification. In other aspects, the management ofthe graphical objects includes measuring characteristics of thegraphical objects.

In a further aspect, the system to organize anatomically related partsinto structures includes a workflow system that receives a plurality ofimages and at least one user input from an external source. The workflowsystem includes two modules or components. One of the two modulesprovides manual or an automated contouring of the anatomical regions andorgans on images in accordance with the user input that ultimatelyyields graphical objects. The manual contouring is performed either bytracing or by follow-up techniques. The automated contouring isperformed either by thresholding or by organ segmentation that has atechnical effect of being considerably faster and more precise thanconventional manual techniques. The other module provides organizationto the graphical objects by creating explicitly or implicitly,containers or group of containers, in accordance with the user inputthat ultimately yields organized containers. A container of a graphicalobject is also known as a structure. The clinicians usually use the nameof the container to identify the graphical object.

In another aspect, the system eases organization of structures byflexible usage of structures. That is, the system creates, stores,retrieves and combines anatomically relevant parts in structures.

In yet another aspect, the systems, method and apparatus described belowis applicable to structure handling from explicit or implicit structurecreation, via drawing graphical object contour either manually (tracing,follow up) or automatically (thresholding, organ segmentation), tostructure management and usage. A segmentation workflow can be used intwo different ways. The first way greatly supports user interaction,intended for organs that are difficult to segment fully automatically(e.g. because of low contrast). Another process supports batch mode,intended for organs whose segmentation is relatively long.

In still another aspect, the systems, method and apparatus describedbelow provide easy-to-use workflow in the correct order. The systems,method and apparatus described below elevates abstraction level,provides consistent organization with clean layout, while allowing auser to maintain control during the segmentation process with a largenumber of choices and options.

In a further aspect, a method to organize anatomically related partsinto structure groups includes creating a plurality of graphical objectsof related anatomical regions and organs, and combining the plurality ofstructures of the graphical objects of the related anatomical regionsand organs.

In yet a further aspect, a method to manage groups of structures ofincludes creating a plurality of graphical objects from predefined dataand creating graphical objects from user-defined data.

Systems, clients, servers, methods, and computer-readable media ofvarying scope are described herein. In addition to the aspects andadvantages described in this summary, further aspects and advantageswill become apparent by reference to the drawings and by reading thedetailed description that follows.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an overview of a system to organizeanatomically related parts into structures;

FIG. 2 is a flowchart of a method to organize anatomically related partsinto structure groups according to an embodiment;

FIG. 3 is a flowchart of a method to organize anatomically related partsinto structure groups according to an embodiment;

FIG. 4 is a flowchart of a method to manage groups of structuresaccording to an embodiment;

FIG. 5 is a block diagram of a hardware and operating environment inwhich different embodiments can be practiced;

FIG. 6 is a block diagram of an hierarchical structure for use in animplementation;

FIG. 7 is a graphical display of the relationship of a number ofanatomical regions;

FIG. 8 is a graphical user interface that illustrates adding a structureon the hierarchical structure tree;

FIG. 9 is a graphical user interface that illustrates linking astructure on the hierarchical structure tree;

FIG. 10 is a graphical user interface that illustrates unlinking astructure on the hierarchical structure tree;

FIG. 11 is a graphical user interface that illustrates a variety ofstructure link functions on the hierarchical structure tree; and

FIG. 12 is a block diagram that illustrates linked structures on thehierarchical structure tree.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description, reference is made to theaccompanying drawings that form a part hereof, and in which is shown byway of illustration specific embodiments, which may be practiced. Theseembodiments are described in sufficient detail to enable those skilledin the art to practice the embodiments, and it is to be understood thatother embodiments may be utilized and that logical, mechanical,electrical and other changes may be made without departing from thescope of the embodiments. The following detailed description is,therefore, not to be taken in a limiting sense.

The detailed description is divided into five sections. In the firstsection, a system level overview is described. In the second section,embodiments of methods are described. In the third section, the hardwareand the operating environment in conjunction with which embodiments maybe practiced are described. In the fourth section, particularimplementations are described. Finally, in the fifth section, aconclusion of the detailed description is provided.

Overview

FIG. 1 is a block diagram of an overview of a system to organizeanatomically related parts into structures. System 100 solves the needin the art for more efficient methods and apparatus of managinggraphical objects. System 100 also solves the need in the art to reducethe challenge to humans in managing graphical objects. System 100 alsoreduces time and improves precision of human clinicians in segmentinganatomical regions.

System 100 includes a workflow system 102 that receives a plurality ofimages 104 and at least one user input 106 from an external source. Theworkflow system 102 includes two modules or components. One of the twomodules 108 provides organization to the images 104 by creatingexplicitly or implicitly structures or group of structures in accordancewith the user input 106 that ultimately yields organized structures 110.Thus system 100 solves the need in the art for more efficient methodsand apparatus of managing graphical objects and reducing the challengeto humans in managing graphical objects.

The other module 112 provides manual or an automated contouring of theimages 104 in accordance with the user input 106 that ultimately yieldsgraphical objects 114. The manual contouring is performed either bytracing or by follow-up techniques. The automated contouring isperformed either by thresholding or by organ segmentation that has atechnical effect of being considerably faster and more precise thanconventional manual techniques. Thus system 100 also reduces time andimproves precision of human clinicians in segmenting anatomical regions.

In some embodiments, the structures of system 100 and methods 200-400are organized in a tree structure in accordance with the occurrence ofanatomical regions. One embodiment of such a tree structure is shown inapparatus 600 in FIG. 6.

System 100 is suitable for any kind of image modality such as X-Rayplane film radiography, computed tomography (CT) imaging, magneticresonance imaging (MRI), and nuclear medicine imaging techniques, suchas positron emission tomography (PET) and single photon emissioncomputed tomography (SPECT). System 100 is also suitable for any kind ofsegmentation algorithm.

Some embodiments operate in a multi-processing, multi-threaded operatingenvironment on a computer, such as computer 502 in FIG. 5. While thesystem 100 is not limited to any particular workflow system 102, image104, user input 106, organization function 108, organized structures110, contour function 112 and graphical objects 114 for sake of claritya simplified workflow system 102, image 104, user input 106,organization function 108, organized structures 110, contour function112 and graphical objects 114 are described.

Method Embodiments

In the previous section, a system level overview of the operation of anembodiment is described. In this section, the particular methods of suchan embodiment are described by reference to a series of flowcharts.Describing the methods by reference to a flowchart enables one skilledin the art to develop such programs, firmware, or hardware, includingsuch instructions to carry out the methods on suitable computers,executing the instructions from computer-readable media. Similarly, themethods performed by the server computer programs, firmware, or hardwareare also composed of computer-executable instructions. Methods 200-400are performed by a program executing on, or performed by firmware orhardware that is a part of, a computer, such as computer 502 in FIG. 5.

FIG. 2 is a flowchart of a method 200 to organize anatomically relatedparts into structure groups according to an embodiment. Method 200solves the need in the art to reduce the challenge to humans in managinggraphical objects. Method 200 also reduces time and improves precisionof human clinicians in segmenting anatomical regions.

Method 200 includes creating 202 a plurality of graphical objects ofrelated anatomical regions. Method 200 also includes combining 204 theplurality of structures of the graphical objects of the relatedanatomical regions.

Method 200 organizes anatomically relevant parts into structure groups,regardless whether the structure groups are explicitly or implicitlycreated in action 202. In some embodiments, explicit creation of astructure or a group of structures includes adding the structure to anexisting or a newly created structure group. In addition a group ofstructures can also be added to other groups of structures.

In some embodiments, implicit creation of a structure or a group ofstructures is accomplished by an organ segmentation process. One exampleof an organ segmentation process is lung segmentation that automaticallycreates structures and an outline of both the right and left lungs.

In some embodiments, in implicit creation of a structure or a group ofstructures, the user is not required to create a structure explicitlybeforehand using segmentation. A segmentation result is stored in avisual graphical object that is created in response to the segmentation,the structure container of the graphical objects being maintained by astructure handling system. Optionally, the user stores the segmentationresult in an already existing structure. When a visual graphical objectis created in response to the segmentation, a pre-defined set ofproperties (e.g. name, color, type) are referenced during the creation.

Segmentation algorithms may create more than one visual graphical object(e.g. lung: right lung and left lung) simultaneously. The structurecontainer of these graphical objects is stored in a structure group thatis created in response to the segmentation, and that are maintained by astructure group handling system.

Some embodiments of method 200 further include one or more of thefollowing operations on one or more visual graphical object(s) such asmultiple selection, union, join, difference, intersect, delete, margin,set visibility, set transparency and/or set color, in any combination ofoperations.

The organ segmentation process provides delineation of anatomicalregions. Segmentation of organs is based on image features andanatomical information. To adapt to different user needs insegmentation, a workflow of segmentation can perform in two differentmethods. The first method is particularly well suited for organs thatare difficult to fully segment without human interaction, as the resultof image problems such as low image contrast. The first process supportsbatch mode via usage of segmentation protocols, which is particularlywell-suited for organs whose segmentation is relatively long. Batch modemeans processing in the background, that is, interaction from the useris not required.

One general method of segmentation workflow includes, selecting an organto be segmented (in batch mode, several organs can be selected andcollected into a segmentation protocol); an optional action of definingseed points or curves; an optional action of adding seed(s) (somesegmentation algorithms may run without providing any seed, especiallywithin a protocol); accepting user interaction during segmentation, thelevel of interaction depending upon the execution mode of eitherinteractive segmentation or batch mode via protocols; and addingsegmented object(s) to the structure (group) list.

In some embodiments, several types of seeds are provided to start theautomatic segmentation, such as: no seed at all, seed point, curve,segment or region of interest (ROI). Providing the seed can be performedin various ways, such as typing point coordinates, clicking and drawingby the user. In addition, suitable seed(s) may be presented to the user,and then user decides which if any seed to accept as given, or adjustingthe presented seed(s), or the user can provide different one(s).

In some embodiments, selecting segmentation parameters has also numerousalternatives, such as: direct typing, loading previously chosen andsaved parameter set, or selection among options. The segmentationparameters minimize the amount of interaction from the user.

In some embodiments, visual cues are provided to help guide the user indata entry. Examples of the visual cues include drawing a circle aroundthe graphical cursor while user aims to select a seed point for eye ballsegmentation, interactively showing the initial region of segmentation,which can range from a minimal region in case of a simple region growingalgorithm to a roughly-fit model in case of model-based segmentation,and a warning that a seed is not correct (e.g. user accidentally clickedin image background. User has the possibility to modify seed points andcurves prior starting the segmentation algorithm. This includesre-typing coordinates, clicking a new seed point, re-placing the pointby dragging, redrawing completely a curve, editing a curve or re-placingthe curve by dragging, etc.

In some embodiments, cases of new seed selection or existing seedmodification are distinguished. A cue used in distinguishing betweencases of new seed selection or existing seed modification is whethergraphical cursor location is near to an existing seed or not, e.g.whether the user has already clicked 1 seed point and the algorithmneeds 2 seed points. When the user clicks in the vicinity of theexisting point again the system interprets the click as a modificationaction, otherwise the clicking is interpreted as a creation action. Thesize of the vicinity is set to a predetermined default value, but theuser can also define the size of the vicinity an override the defaultvalue.

In some embodiments, interactive segmentation provides flexibility andcontrol while maintaining efficiency and ease of use to the user. Ininteractive segmentation, a progress bar displays advancement of thesegmentation process in time. In addition, a “cancel” will cancel theinteractive segmentation when clicked by the user. When the interactivesegmentation is cancelled, the temporary result becomes the finaleresult, which is helpful to an impatient user who is willing to acceptthe current state of the precision of interactive segmentation. The useris able to continue the operation after a segmentation process iscancelled, with the same or different set of segmentation parameters.The balance between precision and speed can be also adjusted as a userpreference.

In some embodiments, a segmentation process can be paused or broken, andthereafter continued, which allows a user to a review the temporaryresult, interactively correct the results between a break and continueoperation. The interactive correction can be organ-specific (extensionof the on-going segmentation algorithm) or general modification withmanual editing tools. The actions of interactive editing can be undo andredo, which can be performed step by step, or the user can select astate or action from the undo/redo list.

In some embodiments of batch segmentation, several organs are collectedfor one background processing.

In some embodiments, interactive segmentation protocols and batchsegmentations are defined in a variety of ways. In one example, a listof all the organs that need to be automatically segmented is receivedfrom the user.

In another embodiments, the organs are ranked in an optimal executionorder, so that:

1) Pre-processing of segmentation requires minimum execution time(optimization in speed): e.g. Smoothing of image data is performed onlyonce and on the union of ROIs containing the individual organs. As aconsequence, saving time is important in case of neighboring organs,such as liver—right kidney or spleen—left kidney.

2) Suitable seed(s) are located (optimization in interaction). In oneexample, the spinal cord is segmented either with a user-given seedpoint or without any seed (system is able to find a proper seed based oncenter position and intensity value of spine). Then rough position ofother organs is automatically found with respect to position and bendingof spine.

3) Leakage is minimized (optimization in quality). In one example, aftersegmentation of the spinal cord, organs of large size (e.g. liver) aresegmented, because they have less chance to visually leak. Finally smallorgans or organs with complex shape (e.g. kidney) follows, during theirsegmentation the already segmented organs are used as constraints toprevent leakage.

A background segmentation protocol process is started after receiving anindication of start. The background segmentation protocol process isperformed with the following seed options: 1) All necessary seeds forall the organs are received from the user in advance. 2) The seeds onlyfor some “difficult” organs are received from the user in advance; forthe other organs, the seeds are determined from the protocol. 3) Allseeds are determined by and received from the protocol. When the seed isdetermined by the protocol, the user is solicited or queried foroptional confirmation about seeds when segmentation of the actual organis starting.

In some embodiments, execution of segmentation protocol conforms to thefollowing requirements: 1) Organs are segmented in the order defined bythe system. 2) The user is solicited to verify thenaccept/adjust/re-give seed, as requested by the user during protocolstart-up. 3) Different organs are not allowed to overlap each-other,which reduces leaking between neighboring organs. 4) Displaying progressof protocol execution. In a default mode, a progress bar is display andthe user is solicited as to whether temporary results of thesegmentation are to be displayed. 5) The user is solicited to canceleither the actual organ segmentation only or the whole protocol. 6) Theuser is solicited to pause or break the segmentation, after that thetemporary result of that organ will become the finale one, and theprotocol continues with the segmentation of the next organ. Interactivecorrection and continuation (of segmentation of the organ stopped) isnot supported after a break and is not supported during protocolexecution.

FIG. 3 is a flowchart of a method 300 to organize anatomically relatedparts into structure groups according to an embodiment. Method 300solves the need in the art to reduce the challenge to humans in managingvisual graphical objects. Method 300 also reduces time and improvesprecision of human clinicians in segmenting anatomical regions.

Method 300 includes creating 202 a plurality of graphical objects ofrelated anatomical regions and storing 302 the graphical objects to acomputer-accessible media, such as a random-access memory (RAM), aread-only memory (ROM), and/or one or more mass storage devices.

Method 300 also includes retrieving 304 the graphical objects from thecomputer-accessible media and combining 204 the retrieved graphicalobjects of the related anatomical regions.

FIG. 4 is a flowchart of a method 400 to manage groups of structuresaccording to an embodiment. Method 400 solves the need in the art toreduce the challenge to humans in managing visual graphical objects.Method 400 also reduces time and improves precision of human cliniciansin segmenting anatomical regions.

Method 400 includes creating 402 a plurality of structures frompredefined data and creating 404 a structure from user-defined data. Inmethod 400, predefined structures and structure groups (e.g. organs suchas eye, lung, spleen, kidney, liver, abdomen, spinal cord, pelvis, etc.)are provided in the creating 402, but a user has the ability to createnew structures or structure groups in the creating 404. That is, thecapability to define structure (or structure group and add any structureto it) is provided to the user, either using an automatic process ormanual drawing. In some embodiments, the manual drawing was createdmanually by a human user of a graphical software tool on a computer.

Each segmentation algorithm creates a structure group and separates thestructures. In one example relating to the lung, segmentation creates alung group and under the lung group is create two sub-structures:left-lung and right-lung. In another example of the eye, for three seedpoints (1-1 point in each eye ball and a 3^(rd) point at the meeting ofoptic nerves), seven structures are created (left and right eye balls,left and right lenses, left and right optic nerves, and chiasma), allbelonging to the organs of sight. This requires the structure properties(e.g. name, color, type) to be defined previously. Accordingly, defaultstructure properties are provided for and referenced by each process.

In some embodiments, these properties are changeable through a userinterface. Thus, method 400 minimizes user interaction to start thesegmentation and provides a convenient way of classifying relatedstructures.

In some embodiments, software-base drawing tools are placed on a fixedor on a floatable toolbar. Floatable drawing toolbar can be placed on agraphical user interface near a drawing/viewing area. In a fixed toolbarthat contains all of the functionalities, the user can drag a button outof the tool palette and drop the button onto a floating panel. Thefloating panel contains only those tools that the user dropped onto thefloating panel, while all the functionalities are available at theiroriginal place. In case of computer systems having dual monitor support,this floatable toolbar is moveable to the second screen as well. In someembodiments of the toolbar, when the user changes the selected structureduring drawing, then the previously used drawing tool remains active. Inthe case of computer systems that are operable to support a plurality ofmonitors or display devices, the floatable toolbar is moveable to anyone of the plurality of monitors or display devices as well.

A “Review Mode” of the drawing toolbar switches off drawing mode,whereupon, the user does not have permissions to edit the selectedstructure. The drawing toolbar contains a “Create Structure” tool.Invoking the “Create Structure” tool creates a new and empty structurethat is the active structure. The drawing toolbar also contains “Selectstructure” tool, which selects a structure with which to work. When theuser selects a new structure, the selection in the table shall changeand the previous structure is saved.

Some embodiments also include a measurement toolbar for measurementtools such as measure distance, angle, surface or ruler, grid etc.

Some embodiments also support customizations or preference by the user.User has the possibility to set which toolbars shall be visible, e.g. bycheckboxes. Some graphical controls such as instructions, preferences ofan opened (activated) panel are be hideable, in which the user can hidethose controls. In some embodiments, preferences include toolbarsettings, manual drawing settings, automatic outlining, organsegmentation settings and measurement settings, level of precisionversus computation time and/or structure properties. In someembodiments, the system loads the previously saved settings. The usercan select among options where default values are stored as preferenceswith the possibility to return to pre-set values (e.g. child or adultcase, adult being the default; or the level of image contrast or noise,where the ones related to usual acquisition are the default).

In some embodiments, methods 200-400 are implemented as a computer datasignal embodied in a carrier wave, that represents a sequence ofinstructions which, when executed by a processor, such as processor 504in FIG. 5, cause the processor to perform the respective method. Inother embodiments, methods 200-400 are implemented as acomputer-accessible medium having executable instructions capable ofdirecting a processor, such as processor 504 in FIG. 5, to perform therespective method. In varying embodiments, the medium is a magneticmedium, an electronic medium, or an optical medium.

Hardware and Operating Environment

FIG. 5 is a block diagram of a hardware and operating environment 500 inwhich different embodiments can be practiced. The description of FIG. 5provides an overview of computer hardware and a suitable computingenvironment in conjunction with which some embodiments can beimplemented. Embodiments are described in terms of a computer executingcomputer-executable instructions. However, some embodiments can beimplemented entirely in computer hardware in which thecomputer-executable instructions are implemented in read-only memory.Some embodiments can also be implemented in client/server computingenvironments where remote devices that perform tasks are linked througha communications network. Program modules can be located in both localand remote memory storage devices in a distributed computingenvironment.

Computer 502 includes a processor 504, commercially available fromIntel, Motorola, Cyrix and others. Computer 502 also includesrandom-access memory (RAM) 506, read-only memory (ROM) 508, and one ormore mass storage devices 510, and a system bus 512, that operativelycouples various system components to the processing unit 504. The memory506, 508, and mass storage devices, 510, are types ofcomputer-accessible media. Mass storage devices 510 are morespecifically types of nonvolatile computer-accessible media and caninclude one or more hard disk drives, floppy disk drives, optical diskdrives, and tape cartridge drives. The processor 504 executes computerprograms stored on the computer-accessible media.

Computer 502 can be communicatively connected to the Internet 514 via acommunication device 516. Internet 514 connectivity is well known withinthe art. In one embodiment, a communication device 516 is a modem thatresponds to communication drivers to connect to the Internet via what isknown in the art as a “dial-up connection.” In another embodiment, acommunication device 516 is an Ethernet® or similar hardware networkcard connected to a local-area network (LAN) that itself is connected tothe Internet via what is known in the art as a “direct connection”(e.g., T1 line, etc.).

A user enters commands and information into the computer 502 throughinput devices such as a keyboard 518 or a pointing device 520. Thekeyboard 518 permits entry of textual information into computer 502, asknown within the art, and embodiments are not limited to any particulartype of keyboard. Pointing device 520 permits the control of the screenpointer provided by a graphical user interface (GUI) of operatingsystems such as versions of Microsoft Windows®. Embodiments are notlimited to any particular pointing device 520. Such pointing devicesinclude mice, touch pads, trackballs, remote controls, touchscreens,point sticks and other special input devices designed for a specificapplication. Other input devices (not shown) can include a microphone,joystick, game pad, satellite dish, scanner, or the like.

In some embodiments, computer 502 is operatively coupled to a displaydevice 522. Display device 522 is connected to the system bus 512.Display device 522 permits the display of information, includingcomputer, video and other information, for viewing by a user of thecomputer. Embodiments are not limited to any particular display device522. Such display devices include cathode ray tube (CRT) displays(monitors), as well as flat panel displays such as liquid crystaldisplays (LCD's). In addition to a monitor, computers typically includeother peripheral input/output devices such as printers (not shown).Speakers 524 and 526 provide audio output of signals. Speakers 524 and526 are also connected to the system bus 512.

Computer 502 also includes an operating system (not shown) that isstored on the computer-accessible media RAM 506, ROM 508, and massstorage device 510, and is and executed by the processor 504. Examplesof operating systems include Microsoft Windows®, Apple MacOS®, Linux®,UNIX®. Examples are not limited to any particular operating system,however, and the construction and use of such operating systems are wellknown within the art.

Embodiments of computer 502 are not limited to any type of computer 502.In varying embodiments, computer 502 comprises a PC-compatible computer,a MacOS®-compatible computer, a Linux®-compatible computer, or aUNIX®-compatible computer. The construction and operation of suchcomputers are well known within the art.

Computer 502 can be operated using at least one operating system toprovide a graphical user interface (GUI) including a user-controllablepointer. Computer 502 can have at least one web browser applicationprogram executing within at least one operating system, to permit usersof computer 502 to access an intranet, extranet or Internetworld-wide-web pages as addressed by Universal Resource Locator (URL)addresses. Examples of browser application programs include NetscapeNavigator® and Microsoft Internet Explorer®.

The computer 502 can operate in a networked environment using logicalconnections to one or more remote computers, such as remote computer528. These logical connections are achieved by a communication devicecoupled to, or a part of, the computer 502. Embodiments are not limitedto a particular type of communications device. The remote computer 528can be another computer, a server, a router, a network PC, a client, apeer device or other common network node. The logical connectionsdepicted in FIG. 5 include a local-area network (LAN) 530 and awide-area network (WAN) 532. Such networking environments arecommonplace in offices, enterprise-wide computer networks, intranets,extranets and the Internet.

When used in a LAN-networking environment, the computer 502 and remotecomputer 528 are connected to the local network 530 through networkinterfaces or adapters 534, which is one type of communications device516. Remote computer 528 also includes a network device 536. When usedin a conventional WAN-networking environment, the computer 502 andremote computer 528 communicate with a WAN 532 through modems (notshown). The modem, which can be internal or external, is connected tothe system bus 512. In a networked environment, program modules depictedrelative to the computer 502, or portions thereof, can be stored in theremote computer 528.

Computer 502 also includes power supply 538. Each power supply can be abattery.

Apparatus

Referring to FIGS. 6-12, a particular implementations are described inconjunction with the system overview in FIG. 1 and the methods describedin conjunction with FIGS. 2-4.

Some embodiments of the structures of system 100 and methods 200-400 areorganized in a tree structure in accordance with the occurrence of thestructures in anatomy.

FIG. 6 is a block diagram of an hierarchical anatomical object structure600 for use in an implementation. FIG. 6 uses the Unified ModelingLanguage (UML), which is the industry-standard language to specify,visualize, construct, and document the object-oriented artifacts ofsoftware systems. In the figure, a hollow arrow between classesindicates that a child class below a parent class inherits attributesand methods from the parent class.

The hierarchical anatomical object structure 600 includes a structureclass 602 that defines attributes (data) and methods (functions) ofobjects that are instantiated from the hierarchical anatomical objectstructure 600. At least four different child classes depend from thestructure class 602 and inherit the attributes and methods of thestructure class 602; a group structure class 604, an eye structure class606, a lung structure class 608, a liver structure class 610, and akidney structure class 612.

The group structure class 604 defines a structure class of a group ofstructures. The eye structure class 606 defines a structure class havingattributes and functions that represent unique aspects of eye anatomy.The lung structure class 608 defines a structure class having attributesand functions that represent unique aspects of lung anatomy. The liverstructure class 610 defines a structure class having attributes andfunctions that represent unique aspects of liver anatomy. The kidneystructure class 612 defines a structure class having attributes andfunctions that represent unique aspects of kidney anatomy.

If user selects an organ-specific segmentation algorithm when nostructure (or group) selected in the structure table, then there will bea new structure added to the table, if the result of the algorithm isonly 1 structure; otherwise, the generated structures will belong to anewly generated structure group; all based on the predefined settings.

FIG. 7 is a graphical display 700 of the relationship of a number ofanatomical regions. Graphical display 700 is display of the organizedstructures 110 in FIG. 1 created in accordance with the hierarchicalanatomical object structure 600 in FIG. 6 by methods 200 in FIG. 2, 300in FIG. 3 and/or method 400 in FIG. 4 in accordance with graphical userinterfaces 800, 900, 1000 and 1100 below.

Graphical display 700 includes column or fields that indicate astructure name 702, visibility 704, color 706 and structure type 708 ofat least one structure. In the example of graphical display 700, fourstructures are displayed. The first structure 710 shown in graphicaldisplay 700 is a group structure named “Group 1” that has visibilityset, no color set, and no structure type.

The second structure 712 shown in graphical display 700 is a childstructure of the “Group 1” 710 named “Structure 2” that has visibilityset, color set to yellow, and an organ structure type. The thirdstructure 714 shown in graphical display 700 is a child structure of the“Group 1” 710 named “Structure 1” that has visibility set, color set toblue, and an organ structure type. The fourth structure 716 shown ingraphical display 700 is a structure not associated with any otherstructure that is named “Structure3” that has visibility set, color setto yellow, and an organ structure type.

In some embodiments, the child and parent structures 710, 712 and 714are associated via links as discussed below FIG. 12. The structures canbe linked in any one of a number of conventional link techniques, suchas singly linked, doubly linked, recursively linked, and/or circularlylinked lists.

In some embodiments, structures (e.g. instantiated objects of theclasses in the hierarchical anatomical object structure 600) can bemanipulated in a variety of ways through GUI 700. Parameters (e.g. name,visibility, color, type) of any structure or structure group that canmodified. A table of the structures can be sorted based on theparameters and displayed. New structures can be added to an existingstructure group. Existing structures can be deleted from a structuregroup. Empty structure groups can be created and structures added to itlater. Structures can be dragged and dropped between structure groups ona graphical user interface. Graphical displays of structure groups canbe opened or closed upon direction from user. An empty structure can beselected and named from a predefined list of names or named from afree-form text format. Multiple structures or structure groups can beselected. Structure contours can be changed with the drawing tools forall structures in selected group or outside a structure group.Union/join, difference, intersect and/or delete operations can beperformed on a volume of structures or structure groups. When a userselects a segmentation algorithm and a structure group, then thegenerated structure(s) are associated with the selected structure groupand a predefined structure group is added. When a user re-generates thestructure group (re-run of a segmentation algorithm within the samestructure group), then the previously deleted structures are generatedagain. When a user executes a segmentation algorithm that alreadyexecuted, then the created structure names will differ. For example,adding a number to the end of the name. e.g.: lung segmentation,left_lung1 and right_lung1. Differentiator characters can be overwrittenor appended at the end of groups and structures.

FIG. 8 is a graphical user interface (GUI) 800 reflecting the treestructure of the hierarchical anatomical object structure 600 in FIG. 6that illustrates adding a structure.

At the highest menu level of GUI 800, three options are presented to theuser; “add structure group” 802, “add structure” 804, and “structures”806. Selecting “add structure group” 802 invokes instantiation of anobject of the group structure class 604 of FIG. 6. Selecting “addstructure” 804 invokes instantiation of an object of the structure class602 of FIG. 6.

Selecting “structures” 806 invokes display of a second level menu of GUI800, which presents four options to the user, “eye” 808, “lung” 810,“liver” 812 and “kidney” 814. Selecting “eye” 808 invokes instantiationof an object of the eye structure class 606 in FIG. 6. Selecting “lung”810 invokes instantiation of an object of the lung structure class 608in FIG. 6. Selecting “liver” 812 invokes instantiation of an object ofthe liver structure class 610 in FIG. 6. Selecting “kidney” 814 invokesinstantiation of an object of the kidney structure class 612 in FIG. 6.

FIG. 9 is a graphical user interface (GUI) 900 reflecting the treestructure of the hierarchical anatomical object structure 600 in FIG. 6that illustrates linking a structure.

At the highest menu level of GUI 900, at least three options arepresented to the user; “unlink structure” 902, “link structure to” 904,and “linked structures” 906. Selecting “link structure to” 904 invokesdisplay of a second level menu of GUI 900, which presents a list of theexisting structures to which a structure can be linked. The example ofGUI 900 displays a list of two structures, “structure1” 908,“structure2” 910 with other structures such as “structure n” 912.

Selecting “structure1” 908 invokes linking of a structure to the“structure1” structure. Selecting “structure2” 910 invokes linking of astructure to the “structure2” structure.

FIG. 10 is a graphical user interface (GUI) 1000 reflecting the treestructure of the hierarchical anatomical object structure 600 in FIG. 6that illustrates unlinking a structure.

At the highest menu level of GUI 1000, at least three options arepresented to the user; “unlink structure” 902, “link structure to” 904,and “linked structures” 906. Selecting “unlink structure” 902 invokesdisplay of a second level menu of GUI 1000, which presents a list of theexisting structures that can be unlinked. The example of GUI 1000displays a list of three structures, ALL 1002, “structure1” 1004 and“structure2” 1006.

Selecting ALL 1002 invokes unlinking of all structures. Selecting“structure1” 1004 invokes unlinking of the “structure1” structure.Selecting “structure2” 1006 invokes unlinking of a structure to the“structure2” structure.

FIG. 11 is a graphical user interface (GUI) 1100 reflecting the treestructure of the hierarchical anatomical object structure 600 in FIG. 6that illustrates a variety of structure link functions.

At the highest menu level of GUI 1100, at least three options arepresented to the user; “unlink structure” 902, “link structure to” 904,and “linked structures” 906. Selecting “linked structure” 906 invokesdisplay of a second level menu of GUI 1100, which presents a list of thestructure functions. The example of GUI 1100 displays a list of threefunctions, “show all linked” 1102, “hide all linked” 1104 and “selectall linked” 1106.

Selecting “show all linked” 1102 invokes display of all linkedstructures. Selecting “hide all linked” 1104 invokes non-display of alllinked structures. Selecting “select all linked” 1106 invokes selectingall linked structures.

FIG. 12 is a block diagram of structure architecture 1200 of treestructure of the hierarchical anatomical object structure 600 in FIG. 6that illustrates linked structures. Structure architecture 1200 showsrelated manifests of an anatomical part on different images linkedtogether, showing an embodiment in which the different graphical objectshave been created and stored, while tracking changes between differentmodalities or between different phases.

In structure architecture 1200, a group of graphical objects (and theircontainer structures) 1202, 1204 and 1206 that are related manifests ofthe same (group of) anatomical region(s) or organ(s) at differentmodalities or phases have linked status through links 1208 and 1210 toeach other and to a reference image series 1212 through links 1214, 1216and 1218.

In some embodiments, the linked status is created upon instantiation ofthe structures. In some embodiments, structures and/or a group ofstructures are manually linked or unlinked at the direction of the useras shown in GUI 900 and GUI 1000 above.

Thus, structure architecture 1200 supports easy follow-up of structuresor group of structures through multiple modalities and phases byproviding the “linked” status in a flexible way.

Structure architecture 1200 is operable for multi-modality and/ormulti-phase structure handling. In some embodiments, multi-modality issupported as follows: One image series has a unique role, it is called“reference modality image series” such as reference image series 1212.An image series having any kind of modality can be selected as referenceimage series 1212. A user contours (both manually and automatically)anatomical regions 1202, 1204 and 1206 on a reference image 1212, suchas a CT image. The contours are automatically transferred to other fusedimages (e.g. MR or PET images). During automatic transformation, theeffect of difference in resolution between modalities (e.g. CT v.s. PET)is automatically corrected.

The automatically transferred structures will be created in the samestructure group as the reference structure belongs to (if any). The usercan inhibit automatic transfer. For example, the user contours andvisualizes specifically pelvic bones on CT only, and bladder on MR only.Optionally, the user draws the contours of the same anatomical region ororgan on any other image, independently of the contours drawn on thereference image. Meanwhile, changing between modalities is allowed. Theuser can modify the automatically transferred contours, which does notaffect the linked status.

When the user directs modification of the contours on the referenceimage 1212, the corresponding contours are optionally aligned on theother (fused) images 1202, 1204 and 1206. The transferred structure orstructure group name will differ (e.g. add a reference to modality tothe end of the name. e.g.: left_lung_CT and right_lung_CT as wellleft_lung_PET and right_lung_PET).

The user optionally names the structure or structure group of the sameanatomical region or organ, drawn independently on the differentmodalities, in a way reflecting their relationship (e.g. left_lung_CTand and left_lung_PET). After that these structures have linked status.Optionally, a differentiator character is overwritten or appended at theend, for groups and structures as well. This will not affect the linkedstatus.

During a “Modality Review Mode,” all selected structures such as 1202,1204 and 1206 that are linked to the reference image 1212 are displayed,regardless on which image the graphical object contours were created. Inaddition, contours and the contained regions of the graphical object canbe shown in review mode.

In some embodiments, color codes are displayed to indicate how muchpercentage of regions is common. As an example, red would indicatecommon regions for most of the contours, orange, yellow, greendecreasing common regions, while blue would indicate region only part ofone structure.

The operations of union/join, difference and intersection on the volumeof selected structures or structure groups are also performed at thedirection of the user. The resulting graphical object of the referenceimage is displayed. In one example, this is useful in comparingdifferences in a bladder CT or MR image. Another option is to apply anyof the previously listed operations to linked structures only.

Common operations (e.g.: show/hide contour, select all) can also beperformed on the linked structures. A user can direct unlinking of onlyone structure from a linked group of structures. A user can directunlinking (i.e. breaking) all connections in the linked group ofstructures. The user can also direct linking a new structure to thereference structure later.

In some embodiments, multi-phase is supported as follows: One imageseries is a “reference phase image series” such as reference imageseries 1212. An image series having any order number of phase in atemporal sequence can be selected as reference image series.

A user contours graphical objects (both manually and automatically) on areference phase (e.g. phase when contrast just injected). The contoursare transferred to the other image phases. During the automatictransformation, misplacement between phases (e.g. due to breathing) iscorrected. The transferred structures are created in the same structuregroup that the reference structure belongs to (if any).

The user can direct inhibition of the transfer. For example, the userdirects contouring and display of a specific phase only. Optionally,user draws the contours of the same object or organ on any phase,independently of the contours drawn on the reference phase. Meanwhile,changing between phases is allowed. The user can also directmodification of the transferred contours. This modification has noeffect on the linked status. When the user directs modification of thecontours on the reference phase, the system aligns the correspondingcontours on the other phases, as directed by the user. The transferredstructure or structure group name will differ (e.g. add a reference tophase to the end of the name. e.g.: left_lung_ph10 and right_lung_ph10).The user can direct naming of the structure or structure group of thesame object or organ, drawn independently on the different phases, in away reflecting their relationship (e.g. left_lung_ph10 and andleft_lung_ph20). Thereafter, these structures will have a linked status.After that these structures have linked status. Optionally, adifferentiator character is overwritten or appended at the end, forgroups and structures as well. This will not affect the linked status.

An image is generated that has intensity values as Maximum IntensityProjections (MIP) of the selected phases. The user can select anycombination of phases to be a basis of MIP phase.

During a “MIP Review Mode,” all selected structures such as 1202, 1204and 1206 that are linked to the reference image 1212 are displayed,regardless of which phase the structure contours were created. Inaddition, contours and the contained regions can be shown in reviewmode.

In some embodiments, color codes are displayed to indicate whatpercentage of regions is common. As an example, red would indicatecommon regions for most of the contours, orange, yellow, greendecreasing common regions, while blue would indicate region only part ofone structure.

The operations of union/join, difference and intersection on the volumeof selected structures or structure groups are also performed at thedirection of the user. The resulting graphical object of the referenceimage is displayed. In one example, this is useful in comparingdifferences in a bladder CT or MR image. Another option is to apply anyof the previously listed operations to linked structures only.

Common operations (e.g.: show/hide contour, select all) can also beperformed on the linked structures. A user can direct unlinking of onlyone structure from a linked group of structures. A user can directunlinking (i.e. breaking) all connections in the linked group ofstructures. The user can also direct linking a new structure to thereference structure later.

The structure and contour handling system preferences are also extendedwith multi-modality settings (e.g. preferred reference image type) andmulti-phase settings (e.g. preferred reference phase type).

In some embodiments of the contouring, various ways of manual drawing(tracing or point-to-point click) and editing can be performed byinteractively modifying (correcting) manual outlines to attach to nearbyvisible borders i.e. gray value differences. In some embodiments,interpolation between contours, regardless if the contours are drawn onthe same type of slices (e.g. only on axial slices) or any mix of slices(e.g. axial/sagittal/coronal slices). In some embodiments, contours arecopied from a previous slice or a template is provided either in 2D or3D forms such circle, ellipse, rectangle, sphere, ellipsoid, etc. Anoutline is created by either thresholding or implementing morphologicaloperators (e.g, remove bridges, fill holes, largest components,margins).

Apparatus components can be embodied as computer hardware circuitry oras a computer-readable program, or a combination of both. In anotherembodiment, the system, apparatus and methods are implemented in anapplication service provider (ASP) system.

More specifically, in the computer-readable program embodiment, theprograms can be structured in an object-orientation using anobject-oriented language such as Java, Smalltalk or C++, and theprograms can be structured in a procedural-orientation using aprocedural language such as COBOL or C. The software componentscommunicate in any of a number of means that are well-known to thoseskilled in the art, such as application program interfaces (API) orinterprocess communication techniques such as remote procedure call(RPC), common object request broker architecture (CORBA), ComponentObject Model (COM), Distributed Component Object Model (DCOM),Distributed System Object Model (DSOM) and Remote Method Invocation(RMI). The components execute on as few as one computer as in computer502 in FIG. 5, or on at least as many computers as there are components.

CONCLUSION

A management system of three dimensional graphical objects is described.Although specific embodiments have been illustrated and describedherein, it will be appreciated by those of ordinary skill in the artthat any arrangement which is calculated to achieve the same purpose maybe substituted for the specific embodiments shown. This application isintended to cover any adaptations or variations. For example, althoughdescribed in object-oriented terms, one of ordinary skill in the artwill appreciate that implementations can be made in a procedural designenvironment or any other design environment that provides the requiredrelationships.

The systems, method and apparatus described above is a complex system,yet is an efficient and user-friendly system, which manages organizationof structures, graphical objects and manual/automated contouring(segmentation). The systems, method and apparatus described above aresuitable for any kind of image modality and any kind of segmentationalgorithm.

This system eases organization of graphical objects and structures byflexible usage of structure groups. That is, the system creates, stores,retrieves and combine anatomically relevant parts.

The systems, method and apparatus described above is applicable tostructure handling from explicit or implicit creation of visualgraphical object of anatomical regions, via drawing the contour of theanatomical region either manually (tracing, follow up) or automatically(thresholding, organ segmentation), to structure management and usage. Asegmentation workflow can be used in two different ways. The first wayhighly supports user interaction, intended for organs that are difficultto segment fully automatically (e.g. because of low contrast). Anotherprocess supports batch mode, intended for organs whose segmentation isrelatively long.

The systems, method and apparatus described above provide easy-to-useworkflow in the correct order. The systems, method and apparatusdescribed above elevates abstraction level, provides consistentorganization with clean layout, while allowing a user to maintaincontrol during the segmentation process with a large number of choicesand options.

In particular, one of skill in the art will readily appreciate that thenames of the methods and apparatus are not intended to limitembodiments. Furthermore, additional methods and apparatus can be addedto the components, functions can be rearranged among the components, andnew components to correspond to future enhancements and physical devicesused in embodiments can be introduced without departing from the scopeof embodiments. One of skill in the art will readily recognize thatembodiments are applicable to future communication devices, differentfile systems, and new data types.

The terminology used in this application is meant to include allobject-oriented, database and communication environments and alternatetechnologies which provide the same functionality as described herein.

We claim:
 1. A tangible and non-transitory computer-accessible medium toorganize anatomically related parts, the medium comprising: a workflowsystem operable to receive a medical image and at least one user inputfrom an external source, the medical imaging comprising one of an X-Rayplane film radiography medical image, a computed tomography image, amagnetic resonance medical image, a nuclear medicine medical image, apositron emission tomography medical image, a single photon emissioncomputed tomography medical image; associating structures with graphicalobjects and contoured anatomical regions; a first component providingcontainers to a plurality of graphical objects of related anatomicalregions that yield structures from explicit or implicit structurecreation in accordance with the user input, the first component yieldingorganized structures, the first component operably coupled to theworkflow system; a second component receiving an outline of anatomicalregions that yield the graphical objects and providing graphical objectcreation and contouring of anatomical regions from the medical image inaccordance with the user input, the second component yielding graphicalobjects, the second component operably coupled to the workflow system;and a third component associating the organized structures withgraphical objects and the contoured anatomical regions, the thirdcomponent operably coupled to the first component and the secondcomponent.
 2. The computer-accessible medium of claim 1, wherein thecontouring of the anatomical regions further comprises: manualcontouring.
 3. The computer-accessible medium of claim 1, wherein thecontouring of the anatomical regions further comprises: automatedcontouring.
 4. The computer-accessible medium of claim 1, wherein theexecutable instructions capable of directing the processor to performthe creating further comprise executable instructions capable ofdirecting the processor to perform: creating explicitly the objectcontainers of the associated anatomical regions.
 5. Thecomputer-accessible medium of claim 1, wherein the executableinstructions capable of directing the processor to perform the creatingfurther comprise executable instructions capable of directing theprocessor to perform: creating implicitly the object containers of theassociated anatomical regions.
 6. The computer-accessible medium ofclaim 1, wherein the executable instructions capable of directing theprocessor to perform the creating implicitly further comprise executableinstructions capable of directing the processor to perform: segmentingthe graphical objects of the anatomical regions.
 7. Thecomputer-accessible medium of claim 1, the medium further comprisingexecutable instructions capable of directing a processor to perform:multiple selecting the graphical objects of the anatomical regions;uniting the graphical objects of the anatomical regions; joining thegraphical objects of the anatomical regions; differentiating thegraphical objects of the anatomical regions; intersecting the graphicalobjects of the anatomical regions; deleting the graphical objects of theanatomical regions; setting visibility of the graphical objects of theanatomical regions; and setting color of the graphical objects of theanatomical regions.